--- title: Calibration Procedure keywords: fastai sidebar: home_sidebar summary: "summary" description: "summary" nb_path: "05_calibrate.ipynb" ---
{% raw %} 
/Users/eway/.pyenv/versions/3.8.3/lib/python3.8/site-packages/pandas/compat/__init__.py:97: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.
  warnings.warn(msg)
{% endraw %} {% raw %}
{% endraw %} {% raw %}
{% endraw %} {% raw %} 
"""
cam_prop.settings["longitude"] = -17.7
cam_prop.settings["latitude"] = 146.1
cam_prop.settings["datetime_str"] = "2021-05-26 03:26"
cam_prop.settings["altitude"] = 0.12
cam_prop.settings["radiosonde_station_num"] = 94299
cam_prop.settings["radiosonde_region"] = "pac"
cam_prop.settings["sixs_path"] = "assets/sixsV1.1"
"""

'\ncam_prop.settings["longitude"] = -17.7\ncam_prop.settings["latitude"] = 146.1\ncam_prop.settings["datetime_str"] = "2021-05-26 03:26"\ncam_prop.settings["altitude"] = 0.12\ncam_prop.settings["radiosonde_station_num"] = 94299\ncam_prop.settings["radiosonde_region"] = "pac"\ncam_prop.settings["sixs_path"] = "assets/sixsV1.1"\n'
{% endraw %} {% raw %} 
HgAr_lines = np.array([404.656,407.783,435.833,546.074,576.960,579.066,696.543,706.722,727.294,738.393,
                           750.387,763.511,772.376,794.818,800.616,811.531,826.452,842.465,912.297])

def sum_gaussians(x:"indices np.array", 
                    *args:"amplitude, peak position, peak width, constant") -> np.array:
    split = len(args)//3
    A   = args[0:split]         # amplitude
    mu  = args[split:2*split]   # peak position
    sigma = args[split*2:-1]    # peak stdev
    c   = args[-1]              # offset
    return np.array( [A[i] * np.exp( - np.square( (x - mu[i])/sigma[i] ) ) 
                        for i in range(len(A))] ).sum(axis=0) + c
{% endraw %} {% raw %}

class SettingsBuilderMixin[source]

SettingsBuilderMixin()

{% endraw %} {% raw %}
{% endraw %} {% raw %}

class SettingsBuilderMetaclass[source]

SettingsBuilderMetaclass(clsname:str, cam_class, attrs) :: type

type(object_or_name, bases, dict) type(object) -> the object's type type(name, bases, dict) -> a new type

{% endraw %} {% raw %}

create_settings_builder[source]

create_settings_builder(clsname:str, cam_class:Camera Class)

Create a SettingsBuilder class called clsname based on your chosen cam_class.

{% endraw %} {% raw %}
{% endraw %}

There are two ways to create a SettingsBuilder class that words for your custom camera. (They involve Python metaclasses and mixins)

For example, you can then create a SettingsBuilder class that words for your custom camera by doing the following.

SettingsBuilder = create_settings_builder("SettingsBuilder",SimulatedCamera)
sb = SettingsBuilder(json_path="assets/cam_settings.json",pkl_path="assets/cam_calibration.pkl")

sb.update_intsphere_fit()
# other calibration functions...

sb.dump()
{% raw %} 
SettingsBuilder = create_settings_builder("SettingsBuilder",SimulatedCamera)
sb = SettingsBuilder(json_path="assets/cam_settings.json",pkl_path="assets/cam_calibration.pkl")
{% endraw %}

Find illuminated sensor area

Since the longer dimension is used for the spectral channels, the rows correspond to the cross-track dimension and are limited by the optics (slit). The useable area is cropped out.

{% raw %}

SettingsBuilderMixin.retake_flat_field[source]

SettingsBuilderMixin.retake_flat_field(show:bool=False)

SettingsBuilderMixin.update_row_minmax[source]

SettingsBuilderMixin.update_row_minmax()

{% endraw %} {% raw %} 
#sb.retake_flat_field()
{% endraw %} {% raw %} 
sb.update_row_minmax()
{% endraw %}

Smile Correction

The emissions lines, which should be straight vertical, appear slightly curved. This is smile error (error in the spectral dimension).

{% raw %}

SettingsBuilderMixin.retake_HgAr[source]

SettingsBuilderMixin.retake_HgAr(show:bool=False)

SettingsBuilderMixin.update_smile_shifts[source]

SettingsBuilderMixin.update_smile_shifts()

{% endraw %} {% raw %} 
#sb.retake_HgAr()
{% endraw %} {% raw %} 
sb.update_smile_shifts()
{% endraw %}

Map the spectral axis to wavelengths

To do this, peaks in the HgAr spectrum are found, refined by curve-fitting with Gaussians. The location of the peaks then allow for interpolation to get the map from array (column) index to wavelength (nm).

{% raw %}

SettingsBuilderMixin.fit_HgAr_lines[source]

SettingsBuilderMixin.fit_HgAr_lines(top_k:int=10)

finds the index to wavelength map given a spectra and a list of emission lines.

{% endraw %} {% raw %} 
sb.fit_HgAr_lines(top_k=16)
{% endraw %}

Each column in our camera frame (after smile correction) corresponds to a particular wavelength. The interpolation between column index and wavelength is slightly nonlinear which is to be expected from the diffraction grating - however it is linear to good approximation. Applying a linear interpolation gives an absolute error of $\pm$3 nm whereas the a cubic interpolation used here gives an absolute error of $\pm$ 0.3 nm (approximately the spacing between each column). Using higher order polynomials doesn't improve the error due to overfitting.

For fast real time processing, the fast binning procedure assumes a linear interpolation because the binning algorithm consists of a single broadcasted summation with no additional memory allocation overhead. A slower more accurate spectral binning procedure is also provided using the cubic interpolation described here and requires hundreds of temporary arrays to be allocated each time. Binning can also be done in post processing after collecting raw data.

{% raw %}

SettingsBuilderMixin.update_intsphere_fit[source]

SettingsBuilderMixin.update_intsphere_fit()

{% endraw %} {% raw %} 
fig = sb.update_intsphere_fit()
{% endraw %} {% raw %}
{% endraw %}

Integrating Sphere data

4D datacube with coordinates of cross-track, wavelength, exposure, and luminance.

Needs testing!

{% raw %} 
exposures  = np.array([0,5,8,10,15,20,50])
luminances = np.array([0,10_000,20_000,30_000,40_000,0])

#@patch
def update_intsphere_cube(self:SettingsBuilderMixin,exposures:np.array,luminances:np.array):
    
    shape = (np.ptp(self.settings["row_slice"]),self.settings["resolution"][1],len(exposures),len(luminances))
    
    lum_buff = CircArrayBuffer(shape[:3],axis=2,dtype=np.int32)
    rad_ref  = CircArrayBuffer(shape,axis=3,dtype=np.int32)
    
    mb = master_bar(range(len(luminances)))
    for i in mb:
        mb.main_bar.comment = f"Luminance = {luminances[i]} Cd/m^2"
        input(f"\rLuminance = {luminances[i]} Cd/m^2. Press enter key when ready...")
        
        for j in progress_bar(range(len(exposures)), parent=mb):
            mb.child.comment = f"exposure = {exposures[j]} ms"
            #self.set_exposure(exposures[j])
            self.start_cam()
            lum_buff.put( self.crop(self.get_img()) )
            self.stop_cam()
        
        rad_ref.put( lum_buff.data )
        mb.write(f"Finished collecting at luminance {luminances[i]} Cd/m^2.")
    
    return xr.Dataset(data_vars=dict(datacube=(["cross_track","wavelength_index","exposure","luminance"],rad_ref.data)),
                                             coords=dict(cross_track=(["cross_track"],np.arange(shape[0])),
                                                      wavelength_index=(["wavelength_index"],np.arange(shape[1])),
                                                      exposure=(["exposure"],exposures),
                                                      luminance=(["luminance"],luminances)), attrs={})

test = update_intsphere_cube(sb,exposures,luminances)
Finished collecting at luminance 0 Cd/m^2.

Finished collecting at luminance 10000 Cd/m^2.

Finished collecting at luminance 20000 Cd/m^2.

Finished collecting at luminance 30000 Cd/m^2.

Finished collecting at luminance 40000 Cd/m^2.

Finished collecting at luminance 0 Cd/m^2.

{% endraw %} {% raw %} 
test
Show/Hide data repr Show/Hide attributes
xarray.Dataset
    • cross_track: 450
    • exposure: 7
    • luminance: 6
    • wavelength_index: 2064
    • cross_track
      (cross_track)
      int64
      0 1 2 3 4 5 ... 445 446 447 448 449
      array([  0,   1,   2, ..., 447, 448, 449])
    • wavelength_index
      (wavelength_index)
      int64
      0 1 2 3 4 ... 2060 2061 2062 2063
      array([   0,    1,    2, ..., 2061, 2062, 2063])
    • exposure
      (exposure)
      int64
      0 5 8 10 15 20 50
      array([ 0,  5,  8, 10, 15, 20, 50])
    • luminance
      (luminance)
      int64
      0 10000 20000 30000 40000 0
      array([    0, 10000, 20000, 30000, 40000,     0])
    • datacube
      (cross_track, wavelength_index, exposure, luminance)
      int32
      2 3 1 1 2 3 1 1 ... 0 0 0 0 0 0 0 0
      array([[[[2, 3, 1, 1, 2, 3],
         [1, 1, 4, 3, 3, 3],
         [3, 1, 1, 3, 0, 2],
         ...,
         [2, 1, 2, 1, 3, 1],
         [2, 3, 2, 2, 3, 3],
         [2, 3, 1, 1, 2, 2]],

        [[2, 3, 1, 1, 2, 4],
         [1, 2, 4, 3, 3, 3],
         [3, 2, 1, 3, 0, 2],
         ...,
         [3, 1, 2, 1, 3, 1],
         [2, 3, 2, 2, 3, 3],
         [2, 3, 1, 2, 2, 2]],

        [[3, 3, 1, 1, 2, 4],
         [1, 2, 4, 3, 3, 3],
         [4, 2, 1, 3, 0, 2],
         ...,
         [3, 2, 2, 1, 3, 1],
         [2, 3, 2, 2, 3, 3],
         [2, 3, 1, 2, 2, 2]],

        ...,

        [[0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         ...,
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0]],

        [[0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         ...,
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0]],

        [[0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         ...,
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0]]],


       [[[2, 1, 2, 2, 1, 2],
         [1, 0, 1, 2, 3, 1],
         [2, 0, 3, 1, 0, 2],
         ...,
         [0, 1, 1, 2, 2, 0],
         [2, 1, 3, 1, 0, 3],
         [2, 2, 0, 0, 3, 3]],

        [[2, 1, 2, 2, 1, 3],
         [1, 0, 1, 3, 3, 1],
         [2, 0, 3, 1, 0, 2],
         ...,
         [0, 1, 1, 2, 2, 0],
         [2, 1, 3, 1, 0, 3],
         [2, 2, 0, 0, 4, 3]],

        [[3, 1, 3, 2, 1, 3],
         [1, 0, 1, 3, 3, 1],
         [2, 0, 3, 1, 0, 2],
         ...,
         [0, 1, 1, 2, 2, 0],
         [2, 1, 4, 1, 0, 4],
         [2, 2, 0, 0, 4, 3]],

        ...,

        [[0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         ...,
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0]],

        [[0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         ...,
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0]],

        [[0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         ...,
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0]]],


       [[[3, 1, 0, 0, 2, 3],
         [0, 3, 3, 1, 0, 0],
         [0, 1, 2, 0, 0, 3],
         ...,
         [0, 2, 0, 1, 3, 2],
         [0, 1, 3, 1, 2, 0],
         [0, 3, 3, 1, 1, 2]],

        [[3, 2, 0, 0, 2, 3],
         [0, 3, 3, 1, 0, 0],
         [0, 1, 2, 0, 0, 3],
         ...,
         [0, 2, 0, 1, 4, 2],
         [0, 1, 3, 1, 2, 0],
         [1, 3, 3, 1, 1, 2]],

        [[3, 2, 0, 0, 2, 3],
         [0, 3, 4, 1, 0, 0],
         [0, 1, 2, 0, 0, 3],
         ...,
         [0, 2, 0, 1, 4, 2],
         [0, 1, 3, 1, 2, 0],
         [1, 3, 3, 1, 1, 2]],

        ...,

        [[0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         ...,
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0]],

        [[0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         ...,
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0]],

        [[0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         ...,
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0]]],


       ...,


       [[[0, 3, 0, 2, 0, 1],
         [1, 2, 1, 3, 1, 3],
         [1, 0, 2, 3, 0, 2],
         ...,
         [1, 3, 2, 2, 2, 1],
         [1, 1, 0, 2, 3, 3],
         [3, 3, 1, 3, 0, 1]],

        [[0, 3, 0, 2, 0, 1],
         [1, 2, 1, 3, 1, 3],
         [1, 0, 2, 3, 0, 2],
         ...,
         [1, 3, 2, 2, 2, 1],
         [1, 1, 0, 2, 3, 3],
         [4, 3, 1, 3, 0, 1]],

        [[0, 3, 0, 2, 0, 1],
         [1, 2, 1, 3, 1, 3],
         [1, 0, 2, 3, 0, 2],
         ...,
         [1, 3, 2, 2, 2, 1],
         [1, 1, 0, 2, 3, 3],
         [4, 3, 1, 3, 0, 1]],

        ...,

        [[0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         ...,
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0]],

        [[0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         ...,
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0]],

        [[0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         ...,
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0]]],


       [[[3, 3, 2, 3, 0, 0],
         [1, 2, 2, 0, 4, 3],
         [1, 2, 1, 1, 0, 3],
         ...,
         [3, 4, 0, 0, 3, 3],
         [2, 2, 3, 1, 2, 3],
         [3, 1, 3, 1, 1, 1]],

        [[3, 3, 2, 3, 0, 0],
         [1, 2, 2, 0, 4, 3],
         [1, 2, 1, 1, 0, 3],
         ...,
         [3, 4, 0, 0, 3, 3],
         [2, 2, 3, 1, 2, 3],
         [4, 1, 3, 1, 2, 1]],

        [[3, 3, 2, 4, 0, 0],
         [1, 2, 2, 0, 4, 3],
         [1, 2, 1, 1, 0, 3],
         ...,
         [3, 4, 0, 0, 3, 3],
         [2, 2, 3, 1, 2, 3],
         [4, 1, 3, 1, 2, 1]],

        ...,

        [[0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         ...,
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0]],

        [[0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         ...,
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0]],

        [[0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         ...,
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0]]],


       [[[3, 0, 1, 1, 3, 0],
         [1, 0, 0, 3, 3, 0],
         [1, 2, 2, 0, 0, 3],
         ...,
         [1, 0, 2, 1, 0, 3],
         [1, 3, 0, 0, 3, 0],
         [3, 3, 2, 0, 3, 2]],

        [[3, 1, 1, 2, 3, 0],
         [1, 0, 0, 3, 3, 0],
         [1, 2, 2, 0, 0, 3],
         ...,
         [1, 0, 2, 1, 0, 3],
         [1, 3, 0, 0, 3, 0],
         [3, 3, 2, 0, 3, 2]],

        [[3, 1, 1, 2, 3, 0],
         [1, 0, 0, 3, 3, 0],
         [1, 2, 2, 0, 0, 3],
         ...,
         [1, 0, 2, 1, 0, 3],
         [1, 3, 0, 0, 3, 0],
         [4, 4, 2, 0, 3, 2]],

        ...,

        [[0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         ...,
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0]],

        [[0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         ...,
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0]],

        [[0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         ...,
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0]]]], dtype=int32)
{% endraw %}

Ignore the duplicated luminance value of 0 to plot. It is not entirely dark since SimulatedCamera is generating random RGB numbers (and applying a red, green, and blue envelope).

{% raw %} 
test.datacube[...,:-1].plot(y="cross_track", x="wavelength_index", col="exposure", row="luminance",cmap="gray")

<xarray.plot.facetgrid.FacetGrid at 0x139e74f40>
{% endraw %} {% raw %} 
f"rad_ref is {test.datacube.size/1024/1024 *4:.2f} MB"

'rad_ref is 148.81 MB'
{% endraw %}